Repository navigation
iouring: a close seam and an ownership trap; the stale closes the canary found are tests' own (Refs #1242) - #1245
Merged
Conversation
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. CodeRabbit Verify
|
…mber closed twice, and the exit paths pinned with it (Refs #1242) Every close of an engine-owned descriptor goes through closeFD/closeFDErr and fdCloseSeam. The trap dup3s a placeholder over each closed number, so a later close of a number still naming it is a stale close whatever else runs. The worker exit paths (cancel, pause and resume, Goexit, Goexit in the pause's close, ring setup failure, first submit failure, bind failure, a panic) close each number once. The stale closes the canary saw are two tests' own: leaveGroupAfterExit's unit test cleans up with an open-check, and the shutdown gate tests' rig closes what a pass closed. Both fail here.
…ot (Refs #1242) Seven tests closed a number the engine had already closed: the cleanup of TestLeaveGroupAfterExitClosesTheRingAndTheListener884 (an open-check, then Ring.Close and close(2) again), the shutdown gate rig and TestGatePassWaitsForTheKernelSendQueue884, and the close-accounting, unregister-close, transplant egress, refused-adopt and wake-eventfd tests. Once a number is free the next open in the process takes it, so the second close destroys whatever that is: the canary's sockets, a sibling test's engine still shutting down, the Go runtime's own. A test now closes a number only while it still names the file it was taken for (fdID1242).
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
FumingPower3925
force-pushed
the
fix/iouring-goexit-double-close-1242
branch
from
November 3, 2026 12:10
63bc9b1 to
9c5fe75
Compare
…il-state close properly, and list the hijack close beside wakefd (Refs #1242) A trap that has been removed closes plainly, so a worker an earlier test leaked cannot dup3 a placeholder that is gone onto a reused number. The ring of the leave-group test is judged by the worker's own state and by what the number names (an io_uring file), not by an inode that rings may share. The nil-state close test fails when finishClose leaves the descriptor open. The unregister-close test no longer dup3s over a number that something else took. The seam's comment names the hijack close as outside it.
… instead of dialing once (Refs #1242) ResumeAccept returns before the workers have re-created their listeners (they do it on their next iteration), so the dial that followed the pause-resume sub-test's resume was refused whenever the worker was slow to run: always on one CPU shared with other work under -race -covermode=atomic (Coverage, run 41638098277). Every dial of these tests now retries to a 15 s bound, and the polls for an exit, a lost listener or a free port are bounded waits, not fixed windows; a premise that is still false then fails with what was seen.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #1242
What the issue guessed, and what is true
The stale-close canary (FLAKES13) printed its hit in the
-vlog duringTestGoexitedWorkerLeavesTheGroupAndThePauseDoesNotWaitForIt848. That test runs its worker in a child process (child848); the parent only waits for it, and prints the child's output only when the child fails. So the hit was made by the parent process, in whatever ran just before. Not the worker's Goexit path.This PR finds the closes with a deterministic trap instead of a timing canary, and the result is:
leaveGroupAfterExit,shutdown,stopAccepting,closeListenerAfterDrainorreleaseFailedInit, so there is no engine change.The trap
fdCloseSeam(fd_close_seam_1242.go) is an atomic test seam under every close of an engine-owned descriptor (closeFD,closeFDErr; the sites that calledunix.Closenow call them; one load of a nil pointer in production, likeringWaitSeam). The test trap (fdTrap1242) does not close(2): itdup3s a placeholder pipe end over the number. That closes the file and keeps the number taken, so nothing can re-open it, and a later close of a number that still names the placeholder is the engine closing something it no longer owns, whatever else runs. No timing, no reuse race. The failure names the number and both sites:TestFDTrapFlagsOnlyAStaleClose1242andTestARemovedFDTrapClosesPlainly1242are the trap's own controls (a removed trap closes plainly, for a worker an earlier test leaked) (a second close is reported with both sites; a number released by someone else and opened again is a first close; a number not open is reported).Tests
TestEveryExitPathClosesEachDescriptorOnce1242: one child process per path (cancel with open connections, pause-resume, goexit with two pauses and resumes, goexit in the pause's close, ring setup failure, first submit failure, bind failure, a ring wait that fails for good, a lost listener) andTestPanickedWorkerClosesEachDescriptorOnce1242(the process dies; the child writes each close and fault to stderr, the parent reads them). Green on main by design: they pin the pairing. Mutants confirm they bite: a duplicatedcloseFD(w.listenFD)inleaveGroupAfterExitfailsgoexitand the panic test; droppingw.listenFD = -1inreleaseFailedInitfailsring-setup-failureandinitial-submit-failure(the ring-wait and lost-listener paths were added after the mutant runs and have none of their own), each naming both sites.TestLeaveGroupAfterExitClosesTheRingAndTheListener884and, throughgateRig884,TestGatePassClosesAtMostTheCapPerPass884,TestGatePassSkipsHTTP2AndBusyConns884,TestGatePassWaitsForTheKernelSendQueue884,TestGatePassHoldsAConnWithARecvCQEPosted1104,TestGatePassHoldsAConnWithBytesInTheSocketQueue1104) and GREEN in the second, which makes a test close a number only while it still names the file it was taken for (fdID1242, fstat dev+inode;fdOpen704asked only "is anything open here"). The other five sites have no trap of their own; the package-wide run below is their evidence.Evidence (golang:1.27.3, 2 CPUs, memlock 8 MiB, -race)
unix.Closein the test files routed through the trap, window of 256 placeholders): the only second closes with a known first close are the seven tests' (about 80 in the gate rig alone). No engine site. A first version of the trap also reported about 1,240 second closes with no recorded first close (RegisterConn,closeOpFD,closeFDsInParallel); they are the trap's own: a test registered a number the trap held as a placeholder, soF_DUPFDreturned a second number naming the placeholder, where a real process gets EBADF. The trap now reports a second close only for a number it closed itself (andopFDhas one close perdriverConn, guarded byopFDOpen). After the second commit: none left (7 reports remain, all of closes of made-up numbers, 65537 and MaxInt32, which the tests hand torefuseAdopt/onAcceptedFD; EBADF, nothing double).OPUS_CANARY=1, whole package-race -covermode=atomic, skipping the three tests the notes name): main: 2 stale closes (one afterTestGatePassWaitsForTheKernelSendQueue884, one afterTestLeaveGroupAfterExitClosesTheRingAndTheListener884, which is also the run's only failure); this branch: 0 and the package passes. Targeted,-count=30of the seven tests: main 29 stale closes and 26 failures; this branch 0 and 0, withTestReportWorkerExitLeavesTheGroupBeforeTheFlag884no longer skipped.go test -race -count=1 -v ./internal/engine/iouring/, as ci.yml runs it) passes on this branch with 8 MiB memlock; the two-worker variants of the setup-failure paths run with one worker there and with two in the CIiouringjob.TestDriverRegisterRacingShutdownReleasesEverySocketstays out of a canary run: it compares the whole fd table before and after, which the canary's own sockets change; it has no close of its own to fix.Checked by reading, no test
The base-tier fallback in
Listen(NewRingthenCloseper probe, once each; a failedmmapcloses its ring fd once, ring.go) has no injection point. Two closes are outside the seam by design: the wake eventfd is closed byinternal/wakefd, which retires the number under its lock first (#655), and a hijacked connection's descriptor is closed through the*os.Filethatnet.FileConnwas given (hijackConn). Routing that one throughcloseFDwould leave the file's finalizer to close the number again at a later GC, after it has been reused, which is the bug this PR hunts.Note for #1243
The seam touches
worker.goin the lines that calledunix.Close(no function of the shutdown send-drain look), andshutdown_gate_884_linux_test.goingateRig884andTestGatePassWaitsForTheKernelSendQueue884.